Adding EC2 OTEL Yaml Merging Config Integ Test #1391

Paramadon · 2024-10-14T21:56:29Z

Description of the issue

This pr adds otel yaml merging integ test. We want to make sure that no regression happens in the agent where the yaml merging for the agent stops working. Passing test run: https://github.com/aws/amazon-cloudwatch-agent/actions/run

Passing test

Amazon-cloudwatch-agent-test repo pr: aws/amazon-cloudwatch-agent-test#424

Description of changes

Test workflow

Starts the agent with the given config.json below using StartAgent() function
Calls sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -s -m ec2 -c file:./resources/otel.yaml
Then we do a curl -X POST -H "Content-Type: application/json" -d @metrics.json -i localhost:4318/v1/metrics
where otlp reciever listens to this and sends it over to cloudwatch using the awsemf exporter.
Then confirm the metric is in the console.

Agent configuration (have one metric being collected in order to generate agent yaml to be merged):

{
  "metrics": {
    "metrics_collected": {
      "cpu": {
        "measurement": [
          "cpu_usage_idle"
        ]
      }
    }
  }
}

Agent Yaml:

exporters:
    awscloudwatch:
        force_flush_interval: 1m0s
        max_datums_per_call: 1000
        max_values_per_datum: 150
        middleware: agenthealth/metrics
        namespace: CWAgent
        region: us-west-2
        resource_to_telemetry_conversion:
            enabled: true
extensions:
    agenthealth/metrics:
        is_usage_data_enabled: true
        stats:
            operations:
                - PutMetricData
            usage_flags:
                mode: EC2
                region_type: EC2M
receivers:
    telegraf_cpu:
        collection_interval: 1m0s
        initial_delay: 1s
        timeout: 0s
service:
    extensions:
        - agenthealth/metrics
    pipelines:
        metrics/host:
            exporters:
                - awscloudwatch
            processors: []
            receivers:
                - telegraf_cpu
    telemetry:
        logs:
            development: false
            disable_caller: false
            disable_stacktrace: false
            encoding: console
            level: info
            output_paths:
                - /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
            sampling:
                enabled: true
                initial: 2
                thereafter: 500
                tick: 10s
        metrics:
            address: ""
            level: None
        traces: {}

Otel Yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  awsemf/otel-merging:
    namespace: "CWAgent-testing-otel"
    log_group_name: "CWA"
    dimension_rollup_option: "NoDimensionRollup"
    log_stream_name: "Testing-otel"
    resource_to_telemetry_conversion:
      enabled: true
    version: "0"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [awsemf/otel-merging]

Merged OTEL configuration:

exporters:
    awscloudwatch:
        force_flush_interval: 1m0s
        max_datums_per_call: 1000
        max_values_per_datum: 150
        middleware: agenthealth/metrics
        namespace: CWAgent
        region: us-west-2
        resource_to_telemetry_conversion:
            enabled: true
    awsemf/otel-merging:
        certificate_file_path: ""
        detailed_metrics: false
        dimension_rollup_option: NoDimensionRollup
        disable_metric_extraction: false
        eks_fargate_container_insights_enabled: false
        endpoint: ""
        enhanced_container_insights: false
        imds_retries: 0
        local_mode: false
        log_group_name: CWA
        log_retention: 0
        log_stream_name: Testing-otel
        max_retries: 2
        namespace: CWAgent-testing-otel
        no_verify_ssl: false
        num_workers: 8
        output_destination: cloudwatch
        profile: ""
        proxy_address: ""
        region: ""
        request_timeout_seconds: 30
        resource_arn: ""
        resource_to_telemetry_conversion:
            enabled: true
        retain_initial_value_of_delta_metric: false
        role_arn: ""
        version: "0"
extensions:
    agenthealth/metrics:
        is_usage_data_enabled: true
        stats:
            operations:
                - PutMetricData
            usage_flags:
                mode: EC2
                region_type: EC2M
receivers:
    otlp:
        protocols:
            grpc:
                dialer:
                    timeout: 0s
                endpoint: 0.0.0.0:4317
                include_metadata: false
                max_concurrent_streams: 0
                max_recv_msg_size_mib: 0
                read_buffer_size: 524288
                transport: tcp
                write_buffer_size: 0
            http:
                endpoint: 0.0.0.0:4318
                include_metadata: false
                logs_url_path: /v1/logs
                max_request_body_size: 0
                metrics_url_path: /v1/metrics
                traces_url_path: /v1/traces
    telegraf_cpu:
        collection_interval: 1m0s
        initial_delay: 1s
        timeout: 0s
service:
    extensions:
        - agenthealth/metrics
    pipelines:
        metrics:
            exporters:
                - awsemf/otel-merging
            receivers:
                - otlp
        metrics/host:
            exporters:
                - awscloudwatch
            processors: []
            receivers:
                - telegraf_cpu
    telemetry:
        logs:
            development: false
            disable_caller: false
            disable_stacktrace: false
            encoding: console
            error_output_paths:
                - stderr
            level: info
            output_paths:
                - /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
            sampling:
                enabled: true
                initial: 2
                thereafter: 500
                tick: 10s
        metrics:
            address: ""
            level: None
        traces: {}

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Manually tested this on ec2 instance by running these command with the above configurations:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop=
sudo rm /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.yaml 
sudo rm /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log 
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -s -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -s -m ec2 -c file:/home/ec2-user/test.yaml
sleep 5
cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log 
go run goScript.go //which simply just does a curl to otlp reciever to add metrics

okankoAMZ · 2024-10-16T14:47:20Z

.github/workflows/integration-test.yml

@@ -13,7 +13,7 @@ env:
  ECR_INTEGRATION_TEST_REPO: "cwagent-integration-test"
  CWA_GITHUB_TEST_REPO_NAME: "aws/amazon-cloudwatch-agent-test"
  CWA_GITHUB_TEST_REPO_URL: "https://github.com/aws/amazon-cloudwatch-agent-test.git"
-  CWA_GITHUB_TEST_REPO_BRANCH: "main"
+  CWA_GITHUB_TEST_REPO_BRANCH: "agentOtelYamlMerging"


this needs to be main right?

…entOtelMerging

adding ec2 test for otel merging

5cd1f83

Paramadon requested a review from a team as a code owner October 14, 2024 21:56

Adding yaml merging integration test

fbe9639

Paramadon mentioned this pull request Oct 15, 2024

Adding integ test for agent otel config merging aws/amazon-cloudwatch-agent-test#423

Closed

Paramadon changed the title ~~adding ec2 test for otel merging~~ Adding EC2 OTEL Yaml Merging Config Integ Test Oct 15, 2024

Paramadon mentioned this pull request Oct 15, 2024

Adding integ test for otel yaml merging aws/amazon-cloudwatch-agent-test#424

Merged

Paramadon added 4 commits October 15, 2024 18:42

changing test repo branch name to test

21349fd

changing branch name back to main

b344dbd

making sure test still works

1f31aa4

chaning test repo branch back to main

f85359e

okankoAMZ reviewed Oct 16, 2024

View reviewed changes

Paramadon added 7 commits October 17, 2024 09:45

Merge branch 'main' of github.com:aws/amazon-cloudwatch-agent into ag…

67411fe

…entOtelMerging

doing a git pull

4dbd4a1

removing verifying terraform version

006cfe5

removing verifying terraform version

99e397b

removing terraform version check

943830e

removing cache

0b745c8

fixing ec2 linux test

3a75a81

Paramadon closed this Oct 24, 2024

Paramadon deleted the agentOtelMerging branch December 17, 2024 13:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding EC2 OTEL Yaml Merging Config Integ Test #1391

Adding EC2 OTEL Yaml Merging Config Integ Test #1391

Paramadon commented Oct 14, 2024 •

edited

Loading

okankoAMZ Oct 16, 2024

Adding EC2 OTEL Yaml Merging Config Integ Test #1391

Adding EC2 OTEL Yaml Merging Config Integ Test #1391

Conversation

Paramadon commented Oct 14, 2024 • edited Loading

Description of the issue

Passing test

Description of changes

Test workflow

Agent configuration (have one metric being collected in order to generate agent yaml to be merged):

Agent Yaml:

Otel Yaml

Merged OTEL configuration:

License

Tests

okankoAMZ Oct 16, 2024

Choose a reason for hiding this comment

Paramadon commented Oct 14, 2024 •

edited

Loading